Learning with Dynamic Programming

We consider the role of dynamic programming in sequential learning problems. These problems require deciding which information to collect in order to best support later actions. Such problems are ubiquitous, appearing in simulation, global optimization, revenue management, and many other areas. Dynamic programming offers a coherent framework for understanding and solving Bayesian formulations of these problems. We present the dynamic programming formulation applied to a canonical problem, Bernoulli ranking and selection. We then review other sequential learning problems from the literature and the role of dynamic programming in their analysis. Frequently in operations research and management science we face uncertainty, whether in the form of uncertain demand for goods, uncertain travel times in networks, or uncertainty about which set of parameters best calibrates a simulation model. Often when facing uncertainty we are also offered the opportunity to collect some information that will mitigate uncertainty’s effects. On one hand, collecting information allows us to make better decisions and produce better outcomes. On the other hand, collecting information carries a cost, whether in time or money spent, or in the opportunity cost sacrificed collecting one piece of information when we could have collected another. How can we optimally balance these costs and benefits? One way to formulate such problems is with Bayesian methods (see, e.g., Berger (1985)), in which we begin with a prior probability distribution that describes our subjective belief about how likely each of the many different possible truths are. Dynamic programming (DP) (Bellman, 1954) offers a general way to formulate and solve such Bayesian sequential learning problems. This DP formulation provides value in three ways. First, in some cases, the DP formulation allows computing the optimal solution. This is usually only true for smaller problems (e.g., the inventory problem considered in Lariviere and Porteus (1999)), because the curse of dimensionality (see, e.g., Powell (2007)) prevents computing the solution to larger problems in a practically feasible amount of time. Second, the DP formulation sometimes allows structural results to be shown theoretically, which either provides intuition about the problem and the behavior of the optimal policy as in, for example, Ding et al. (2002), or provides a characterization of the optimal policy that may be directly exploited to find an optimal policy, as in sequential hypothesis testing (Wald and Wolfowitz, 1948). Third and finally, the DP formulation sometimes suggests useful heuristics (e.g., knowledge-gradient methods (Frazier, 2009)) for complex and large-scale learning problems. The sequential learning problems that we consider here have been studied in several separate communities. Within the operations research community, sequential learning problems, as well as the way in which DP can be used to confront them, were recognized in early work by Howard (Howard, 1960, 1966). Within the resulting literature, surveyed in Monahan (1982); Lovejoy (1991), such sequential learning problems were called Partially Observable Markov Decision Processes (POMDP). This work on POMDPs was continued in the artificial intelligence and reinforcement learning communities, (see, e.g., Cassandra et al. (1995); Kaelbling et al. (1998)), where the emphasis is on general purpose complexity results and algorithmic strategies that apply to the whole spectrum of POMDPs. Specific applications tend to come from robotics, game playing, and other problems that are thought to be exemplars for the problems that humans face when interacting in a general way with their environment. More recently, these problems have also been called Bayes